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STEP-UP SIMULTANEOUS TESTS FOR IDENTIFYING ACTIVE 
EFFECTS IN ORTHOGONAL SATURATED DESIGNS 

By Samuel S. Wu and Weizhen Wang 1 

University of Florida and Wright State University 

A sequence of null hypotheses regarding the number of negligible 
effects (zero effects) in orthogonal saturated designs is formulated. 
Two step-up simultaneous testing procedures are proposed to iden- 
tify active effects (nonzero effects) under the commonly used assump- 
tion of effect sparsity. It is shown that each procedure controls the 
experimentwise error rate at a given a level in the strong sense. 

1. Introduction. Assume a linear model 

(1) Yi = n + (3ixa -\ \-(3 k x ik + ei, for i = 1, . . . ,M, 

where £j ~ i.i.d. N(0,cr 2 ). The unknown parameters Pi are of interest and [i 
and a are two unknown nuisance parameters. The design is called orthogo- 
nal if the least squares estimators Pi (1 < i < k) are uncorrelated (equivalent 
to independent), which occurs, for example, in two-level fractional factorial 
designs. The design is said to be saturated if there are just enough observa- 
tions to estimate the model parameters Pi and (i.e., M = k + 1), leaving 
no degrees of freedom to estimate the error variance a 2 . In order to make 
inferences on Pi, one must typically use the assumption of effect sparsity, 
that is, that most of the Pi's are equal to zero. Then we can use the cor- 
responding P^s to estimate a 2 . However, we do not know how many and 
which of the Pi's are zero. An initial guess would be at least v of the Pi's 
equal zero, say one-half of the effects. Therefore, the smallest v of the pf's 
should be used to estimate a 2 . Any other (3j whose square is substantially 
larger is likely to have a nonzero mean and corresponds to an active effect. 
For a fixed sequence of (3 = (Pi , . . . , let 

(2) N = the number of Pi's which equal zero. 
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Thus, the number of nonzero Pi's is equal to k — N and the entire parameter 
space without nuisance parameters is H = {(3 = (Pi, . . . ,Pk) : N > u}. For 
each integer m € \y + 1, k], consider the testing problem 

(3) i?o,m : N > m vs. H A , m ■ N < m - 1 (i.e., k - N > k - m + 1) 
and define a parameter configuration in each i?o,m; 

(4) /3 m =:(0,...,0,+oo,...,+oo), 
where the first m components are zero. Let 

(5) B={H^ m :v+l<m<k], 

which contains all null hypotheses of interest in this paper. Because Hq^ is a 
subset of Hqj for any i> j, if Hqj is incorrect, then so is H$^. This implies 
that a testing process should be terminated as soon as a rejection occurs 
for some null hypothesis. Starting from m = u + 1, we test these hypotheses 
one at a time as m goes up to k. If i?o,^+i is rejected, we then conclude 
that there are k — v active effects (i.e., H n Ha,u+\) and no longer test any 
other hypotheses; otherwise, we test the next hypothesis In general, 

if i/o,m is the nrs t hypothesis being rejected for some uiq < k, we stop and 
conclude that there are k — uiq + 1 nonzero effects (i.e., -ffo,m -i H ^A,m )i 
otherwise, all hypotheses in B are accepted and we conclude that there is 
no active effect. Clearly, this is a step-up testing procedure. 

Many inference procedures have been proposed to identify active effects. 
The data analysis of orthogonal saturated designs was initially considered 
by Birnbaum [1] and Daniel [2]. The half-normal plot introduced by Daniel 
[2] is still being used in the preliminary analysis. Lenth [8] proposed the 
first adaptive method to let the data determine which and how many of 
the P^s should be used to estimate a 2 . Whether Lenth's interval is of level 
1 — a still remains a question. Besides using the data adaptively, another 
fundamentally desirable property is the ability to control the error rate in 
the strong sense (i.e., under all parameter configurations), which is espoused 
by Hochberg and Tamhane [5]. For orthogonal saturated designs, the first 
adaptive confidence interval known to provide strong control of error rates 
and more general results can be found in [15] and [16], respectively. Hamada 
and Balakrishnan [4] provided a thorough review of the analysis methods 
available for saturated designs. 

Since we do not know which and how many effects are active, it is rea- 
sonable to search for active effects using simultaneous tests. Here are two 
possibilities: 

(a) Starting from the largest $f, test whether the corresponding effect is 
active. If it is not active, then conclude no active effect and stop; oth- 
erwise, test for the second largest Pf (go down), and so on, until a zero 
effect is found — the step-down tests. 
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(b) Starting from the (y + l)th smallest /3f, test whether the corresponding 
effect is active. If it is active, then conclude k — v active effects and stop; 
otherwise, test for the (y + 2)th smallest (3f (go up), and so on, until an 
active effect is found — the step-up tests. 

Voss [12] proposed nonadaptive step-down tests for a set of hypotheses 
different from B and controlled the experimentwise error rates at the given 
level q in the strong sense. Recently, Voss and Wang [14] derived adaptive 
step-down tests with the experimentwise error rates controlled in the same 
setting as Voss [12]. As pointed out by several researchers [3, 7, 11], the 
step-up tests typically have a greater power to detect active effects than 
the step-down tests. For example, Venter and Steel [11] proposed one step- 
up and one step-down procedure for the orthogonal saturated designs with 
cutoff points determined at /3 . However, they were unable to prove that 
the experimentwise error rates of their procedures are controlled at a given 
level a in the strong sense. 

The rest of this article is organized as follows. In Section 2, we provide 
the motivation for two desirable tests for i?o,m an d establish a probability 
inequality for noncentral \ 2 distributions which asserts that the maximal 
type I error of any test in a general class is always achieved at (3 . Based 
on this inequality, two level- a tests are proposed in Section 3 for testing 
each single hypothesis Hq^. Two sequential step- up procedures for testing 
all hypotheses in B, which control the experimentwise error rates at a in the 
strong sense, are derived in Section 4. Section 5 presents a simulation study 
and Section 6 concludes with some discussion. 

2. Motivation and a general probability inequality. In this section, we 
provide motivation for the new procedures testing i?o,m and a general class 
[see A m in (11)] of the rejection regions of level-a. The maximal type I error 
of each rejection region in this class is always achieved at j3 , as stated in 
Theorem 1. 

Assume the factorial effect estimators (3i are independently distributed 
as N(f3i,afa 2 ) for known constants Oj. We may assume that each en = 1 
without loss of generality. Let X\,...,Xk be the order statistics of the pf 
for i <k. Intuitively, it is more likely that the small order statistics Xi will 
correspond to estimators f3j with /3j = 0. If we believe a priori that at least 
v of the P^s are zero, then those /Vs corresponding to X±, . .. ,X V are likely 
to be the negligible ones. To test -ffo,m> one needs to compare X m with 
X\ , . . . , X v . For any integer n E [u, m] , let S n = Ya=i ^* anc ^ X n = S n /n and 
define a test statistic for i?o,m as follows: 
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Intuitively, a large value of W n<m is in favor of i?A,m- Therefore, the rejection 
region should be W n , m > d n>m for a constant d n ^ m which satisfies 

(7) sup Pp(W n>m > d n>m ) = a. 

/3S-flo, m 

[ i : this section, we will show that W n , m is stochastically largest at (3 for 

Definition 1 . A random variable X is said to be stochastically smaller 
than Y, denoted by X ~< Y, if P(X <d)> P(Y < d) for all d. 

Let Yi >m , . . . , Y m ^ m be the order statistics of the pf 's for i < m when /3i = 
• • • = Pm = and let 



nY 

( o\ 7 _ nl m,m 

\°) *->n.pi — v 

Langsrud and Nass [7] also studied Z n>m , called \I/ m) o,n,m-distribution in their 
notation. Clearly, the distribution of Z n ^ m does not depend on any param- 
eter and can be sampled based on the order statistics of m independent Xi 
random variables. It is easy to see that W n ^ m = Z n ^ m at (3 and we want to 
show that W n>m -< Z n ^ m on i?o,m- If this is true, then (7) reduces to 

(9) P(Z n ,m > d n>m ) = a 
and d n ,m is the 100(1 — a) percentile of Z njm . 

Definition 2. A function h : R d — » E will be called nondecreasing (to 
the coordinatewise ordering) if X{ < pi, i = 1, . . . , d, implies that h(x) < h(y). 

Now we prove a general theorem that includes (9) as a special case. 

Theorem 1. Suppose that Xi, . . . ,X/. are the order statistics of inde- 
pendent random variables with noncentral chi-squared distributions Xi(Pi)> 
1 < i < k. Then 

(10) sup Pp{R) = Pp (R) 

PeH , m ~ 

for any rejection region R of i?o,m that belongs to the class 

(11) A m = {T L {X 1 ,...,X s - 1 ) <T R {X S ), for some integer 1 < s < m}, 
where and Tr satisfy the following two properties: 

(i) (monotone) Ti(x\, . . . , x s -i) and Tr(x s ) are nondecreasing functions; 
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(ii) (invariant) Tl/Tr is invariant to scale transformation, that is, for 
any a > 0, 

T L (axi, . . .,ax a -l) T L (x 1: . . . ,» s _i) 



(12) 



T R (ax s ) T R (x s 



Corollary 1. For any < a < 1 and d n ^ m given in (9), the rejection 
region 

(13) Rn,m = {W nt m > 

defines a level-a test for (3). 

Proof. Let s = m, T L (x\, . . .,x m -i) = Y!i=\ x il n andr R (x m ) = x m /d n , m . 
The claim follows from Theorem 1. □ 

To prove Theorem 1, we need the facts below (Corollary 2 and Corol- 
lary 3). 

Lemma 1. For a nonnegative random variable X and a positive number 
y, let X y = X/y, given that X < y. Let U ~ Xv a chi-squared distribution 
with one degree of freedom, and V ~Xi(# 2 ), a noncentral chi-squared distri- 
bution with one degree of freedom and noncentrality parameter 6 2 . Then we 
have (i) U y -< V y and (ii) U y2 -< U yi for any yi > y\ > 0. 

Part (ii) of Lemma 1 is identical to Lemma 2 of [7]. The proofs for both 
stochastic orderings follow from the monotone likelihood ratio function. 

Lemma 2. Let U±, . . . , U s -\ be any independent random variables. Let 
the same be true for V\, . . . , V s -i- IfT(x\, . . . , x s -\) is a nondecreasing func- 
tion and Ui<Vi for i < s — 1, then 

T(Ui,...,U 8 -i)*T(Vi,...,V,-i). 

This is called by some researchers (see, e.g., [13]) a "stochastic ordering 
lemma." 

Corollary 2. Suppose that Ui ~ x\ , 1 < « < s — 1, and Vj ~ Xi(6j)> 
1 < j < s — 1 , are independent random variables. For any y > 0, let Uu\ y be 
the order statistics ofUi tV (= U%jy, given that Ui < y, as defined in Lemma 1) 
and let V^^ y be the order statistics of Vj )V . Then for any nondecreasing 
function Tl, 

(14) Tl(U^ v , J7( S _ 1 ) )J/ ) -< T L (V(i)^ y , V( s _i^ y ). 
Therefore, for any t>0, 

P(T L (U {1)j y, U (s _ lly ) <t)> P(T L (V (l) ,y, . . . , V (s _ 1))y ) < t). 
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Proof. Let 

T L(Ul,y, U s -l,y) = T L (U(i)^ y ,. . . , U( s _x),y)- 

Since each Un\y is a nondecreasing function in each Uj tV and Tl is nonde- 
cr easing in each of its arguments, T£ is nondecreasing in each of its argu- 
ments. Therefore, if one combines part (i) of Lemma 1 with Lemma 2, it can 
be concluded that 

t l{U{i)^, . . . , U( s _i^ y ) = Tl(U ljy , Ug-^y) -< Tl(Vi t y, V s ~i >y ) 

= 7l(V(i) i j / ,...,V( s _i) i3/ ). □ 



Corollary 3. Suppose that Tl and Tr satisfy the monotone and in- 
variant conditions specified in the definition of A m in (11) and one defines 

G*(y) = P(T L (U {1)jy , U {s _ 1)>y ) < T R (1)) 

/ ^ IltZl f(ui)duj 

{0<u 1 <---<u.- 1 <y,T L (u 1 ,---,u s -i)<T R (y)} F(y) s ~ l 

Then under the condition of Corollary 2, 2x,([7m j,, . . . , Ur s _i\ y ) is stochas- 
tically nonincreasing in y, that is, for yi > y\ > 0, 

T L(U(i) m ,. • • , U {s _ 1)tV2 ) -< T L (Up) iyi ,. . . , J7( s _i) iW ). 

Therefore, G*(y) is a nondecreasing function. 

Proof. Combine part (ii) of Lemma 1 with Lemma 2 and define T£ as 
in Corollary 2. □ 



Proof of Theorem 1. Consider two samples, 

{/3f,...,/3 m ,+oo,...,+oo (k — m of them)} and {/3j , . . . , fil} . 

Clearly, the sth order statistic of the first sample is stochastically larger 
than that of the second sample, that is, X s -< Y S)Tn for any given integer s 
satisfying 1 < s < m. 
Second, we have 

P(T L (Y hm , . . . ,y s _i, ro ) < T R (Y a , m )) 

= E[P(T L (Y hm , . . . , y a _ ljm ) < T R (y)\Y s , m = y)] 
= E[P(T L (U (lly , l/ (a _i )lV ) < T R (l)\Y., m = y)] 
= E[G*(Y a>m )]. 
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Next, for any partition uj = (ji, j'2, 



gers 1 to A;, we denote by E u the event {/3 2 . < /3 2 ,Vi < s; /3 2 <ffi ,VI>s}- 



i s -i)(is)(is+i,---,Jfe) of the inte- 

£<$.,Vi<,; /3 2 - 



We note that for any (3 E Ho, m , 

= Ep[Pp{T L {X u . . . ,X s _i) < Tr^IX, = y)] 

= Ep[Pp(T L (x liy , . . . ,x a _i, y ) < r fl (i)|x a = i/)] 

^p ya ({r i (x 1 , y , . . .,x s _ 1:2/ ) < Tfl(i)} n ^|x s = y ) 



= Ep 
<Ep 



J2Pp({T L (X 1>y , . . .,X,- 1)y ) < T R (l)}\E u ,X a = y)Pp{E u \X a = y) 

YsPpJlTLiX^y, . . .,X 8 . ltV ) < T R (1)}\X S = y)P^(E UJ \X s = y) 

= Ep[PpjT L (X ltV , . . .,X a - ltV ) < T R (1)\X S = y)} 

= Ep[P(T L (U {lly , U (a . 1)>y ) < T R (1)\X S = y)\ 

= Ep[G*(X s )}, 

where the above inequality is due to Corollary 2. 

Finally, since G*(y) is a nondecreasing function due to Corollary 3, the 
inequality X s -< Y s>m implies that G*(X S ) -< G*{Y s ,m)- Therefore, it can be 
concluded that 



Ef3[G*(X s )}<E[G*(Y s>m )}. 



□ 



3. Two tests for Ho^ m . In this section, we present two tests for -ffo.m- 
While both are of level a, they are to be used under different circumstances. 
Let dy^m and d m _i jm denote the numbers determined by (9) when n = v and 
n = m — 1, respectively. 

Theorem 2. For any < a < 1, the rejection regions 
(15) R V:Tn = {W U)m > d Vtrn } and R m -l,m = {W m 

— l,m d m — \,m} 

both define level-a tests for (3). 

This theorem is a special case of Corollary 1 when n = v and n = m — 1 . 
The test based on R v , m estimates the error variance by X u = Ya=i Xi/u, irre- 
spective of the value of m, while the test based on i? m _i >m uses the adaptive 
estimator X m _i = Yli=i Xi/(m — 1). These are referred to by Venter and 
Steel [11] as fixed and sequential scaling, respectively. Region R vm should 
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be applied only when i?o,m is of interest and no information is available 
on whether iio,n is true for any n <m. In such a case, we are certain that 
at least v of the /3?'s have a zero mean. It is reasonable to compare X m 
with the average of X±, . . . , X v , the smallest v of the ftf's, through their 
average. If W u , m is large, then one concludes that all ftf's corresponding to 
X m , . . . , X/. are from populations with nonzero means. On the other hand, 
if one tests i?o,n for n <m sequentially up to Ho, m and Ho )m -i is accepted, 
then at least m — 1 of the /3j's are zero. In this case, one should compare X m 
with Xi, . . . ,X m -i and a large value of W m -\,m would lead to a rejection 
of -Ho,m- 

4. Two step-up testing procedures. When effect sparsity is assumed, 
we do not know which and how many of the /3,'s are zero. It is of more 
interest to conduct tests simultaneously to identify the nonzero effects. The 
tests developed in the previous section, in fact, can detect whether there is a 
jump at X m among X\, . . . , X^. However, these tests cannot tell whether the 
jump, if it exists, is the first one, which is what interests us. Therefore, as 
mentioned earlier, since Ho,m decreases as m increases, one needs to conduct 
tests sequentially. Like all testing problems, there are two major concerns: 
to control the experimentwise error rate at a given level a, that is, 

sup (assert not -ffo.n, which contains (3, 

(16) ^ULh-i*"."* " 

for some n G [v + 1, k]) < a, 

and to obtain more powerful tests, which means larger rejection regions. 

The first requirement (16) can be ensured by using the closed test pro- 
cedure proposed by Marcus, Peritz and Gabriel [9]. For details, see [6], 
page 137. A naive solution is to assert not i?o,m (i-e., to assert Ha,™,) if 
one rejects at level a for all i>m. For example, suppose R v ^ is used to 
test for each Hq^. Then assert not Ho t7n iff R m = C\i =m R v ,% is true. This, by 
the closed test procedure, controls the experimentwise error rate at a. Note 
that R m decreases as m increases, which contradicts the fact that HQ m is 
decreasing (we need R m to increase). Therefore, simply applying the closed 
test procedure on the tests derived in the previous section only results in 
less powerful tests for the simultaneous hypotheses. We require that the re- 
jection region for HQ m (a) increases as m gets larger and (b) is of level-a. In 
this section, two testing procedures are discussed with their rejection regions 
denoted by {R^mjt^+i and { R m-i,m\m=u+i corresponding to R U)7n and 

Rm— 

i >m , respectively. 
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4.1. The construction of {Rtm}m=u+i' step-up tests with fixed scaling 
(SUF). The general form of Ry m , for v + 1 < m < k, is 

m 

(17) R* Utm = |J {W v , i >%J = {S v <max{uX i /<Z ti }V v+1 }, 

i=v+l 

where the sequence of constants {d*l m } k m=v+ i is determined iteratively be- 
low. More precisely, d* m depends on d* u i for i < m and causes R* m to have 
level-a. It is clear that R* vrn is nondecreasing when m gets larger and is 
strictly increasing if all d* m are finite. We start with the following lemma. 

Lemma 3. For a sequence of random variables {Aj}f =0 , where s is a 
given positive integer, 

s 

(18) P(A < maxjAJti) < E P ( max { A i};=0 < A 0- 

i=l 

Proof. We prove (18) by induction. When s = l, (18) is true. Suppose 
(18) is true for any s = n. Then for s = n + 1, 

P(A < maxjAOg 1 ) < P(A < max{Ai}? =1 ) + P(max{A,}™ =0 < A < A n+1 ) 

n+l 

<^P(max{A i }}lJ<A i ). □ 

i=l 

We now determine the sequence of constants {d* m } k n=u+1 starting from 
m = v + 1. For testing iJo,i>+i, let P^+i = P„^+i. It is a level-a test by 
Theorem 2. 

For any v + 2 < m < k, let d* v = oo and suppose that are 
available. Then d* m is determined by solving 

in 

E P /3 (^)=«. 

r * - — m 

(19) 

where Aj = {max{S„, {vXj /dlj})J v } < iz-Xi/d*^}. 
Note that R* um is of level-a because for any 3 E flo,m> 

m m 

(20) p^(p; m )< E W)< E P /3 (^) = «, 

i=i/+l i=u+l 

where the first inequality follows from Lemma 3 (with Ao = S u and Aj_ y = 
vXijd* vi for i = v + l,...,m) and the second inequality holds since each 
term in the summation achieves its maximum at 3 by Theorem 1. On the 
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other hand, since the thresholds {tC,i}£i/+i satisfy Y^Lj+iPp A\Ai) = a 
and each term in this summation satisfies Pr (At) > Pr (A4) by Theo- 
rem 1, the last term in the summation of (19) is greater than zero, that is, 
Pf3 {Am) > 0. This guarantees d* to be finite, which implies that rejection 
region R* m is larger than R* m _ 1 . 

Finally, for m = k, since the null hypothesis Hq^ now contains only one 
parameter configuration (3 , and d* i is available up to i = k — 1 , one deter- 
mines d* k by solving 

(21) Pp h (RZ tk )=a, 

which implies that is level-a. Similarly, one can show that d* k is fi- 
nite. Thus, R* yk is larger than R* uk _ v The determination of {d* m } m=l/+1 is 
completed. 

To conduct the simultaneous tests for £>, assert not -ffo,m (he., assert 

HA,m) 

(22) if R* v m is true. 

Notice two facts: (1) B is closed under the operation of intersection and 
(2) for each v + 1 < m < k, R* um = f] ] ^ =m R* i is level-a. Therefore, the ex- 
perimentwise error rate is no greater than a by the closed test procedure. 
The discussion above is now summarized as the following theorem. 

Theorem 3. The rejection regions R* um given in (17) increase when m 
gets larger and each defines a level-a test for -fTo.m • If one conducts simulta- 
neous tests for B using (22), then the experimentwise error rate is controlled 
at a in the strong sense. 

Let [1], . . . , [k] be random indices such that Mj < • • • < ffii. We now de- 
scribe the step-up testing procedure based on R* m as follows: 

Step 1: If Rtv+l 1S true, then conclude that P\ v +\\ , . . . , /3ny are the k — v 
active effects (= H D Ha,u+i) and stop; otherwise, go to step 2. 

Step 2: If R„ v +2 1S true, then conclude that P[ v +2] > • • • > P[k] are the k — u — 1 
active effects (= Hq iV+ i n Ha.u+2) and stop; otherwise, go to step 3. 

Step k — v: If R* k is true, then conclude that /3m is the only active effect 
(= -ffo,fc-i H i?o,fe) an d stop; otherwise, conclude no active effect and stop. 

4.2. The construction of {R m _ lm } m=u+1 : step-up tests with sequential 
scaling (SUS). There is another way to conduct the simultaneous tests for 
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B. For each integer m £ [v + l,k], we construct a level-a region for ffo,mj 
denoted by R* m _i m (corresponding to i? m _i jm in Section 3), of the form 

1 j ={5,<max{Q i }- I/+1 } ! 

where Q l = (i- ^Xjd*^ - + 5, and = E}=\ 

To determine constants ft-imlm^+ii we fi rs t let the constant d* u+1 
equal d^^+i. Suppose that d*_ 1 i is available up to i = m — 1 for m < k. We 
then determine d* m _ lm . Comparing (23) with (17), Rm-im an d Rt m have 
similar forms. Therefore, similarly to (19), we obtain dj^_ lm by solving 

m 

(24) P P (max{5„{g j }p 1 t/ }<ft) = « 

i=u+l 

(with Q v = 0). Finally, for m = k, since c^-i m is available up to m = k — 1, 
dfe_i,fc is solved by Pp k (R* k _ lk ) = a. The determination of {#™-i,nJm=H-i 
is thus complete. 

Using a discussion similar to that used for R„ m , one can show that 
R* _± m is a level-a test for -ffo,m and is increasing in m. More specifically, 
Lemma 3 implies that 

m 

< P§fa»x{Sv,{Qj}&}<Qi) 

i = v+l 

The last step rewrites each set and makes it clear that, by Theorem 1, each 
probability above on the right-hand side achieves its maximum at 3 among 
(3 £ -ffo,m- Therefore, the type I error of -R^-i m * s bounded by a due to (24). 
Again, by Theorem 1, each term corresponding to i < m evaluated at 3 is 
smaller than that at 3 , , which ensures the existence of a finite solution 

— m— 1 

for d m _i m . 

To conduct the simultaneous tests for the null hypotheses in B, assert not 
H ,m (i.e., assert H A , m ) 

(25) if iC-i,m is true. 
Therefore, we have a theorem similar to Theorem 3. 

Theorem 4. The rejection regions m gi ven i n (23) are increasing 

when m increases and each defines a level-a test for -ffo,m- If one conducts 
the simultaneous tests for B using (25), then the experimentwise error rate 
is strongly controlled at a. 
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We omit the description of the step-up testing procedure based on R^-im- 

Remark 1. Langsrud and Naes [7] and Venter and Steel [11] also con- 
sidered these two step- up procedures. For the same test statistics W Vjm and 
Wm-i,m ; they proposed to determine critical values &\ m and <$ m _ l m itera- 
tively by 

P P m [ U ><<})=« and pA (J {Wi_ M >4_ M })=a. 

\i=i/+l / \i=i/+l / 

(26) 

Intuitively, the solutions dj, m and <4„-i m to the above equations would be 
smaller than their corresponding cutoff points d* vm and d* m _ x m determined 
by (19) and (24) and would hence result in larger rejection regions. However, 
it is still not clear that the error rates of their procedures are controlled at 
a in the strong sense because it is very difficult to establish that for all 

(27) Pp{K,m)<Pp (K,J or P /3 ( J R^_ 1>m )<P /3 (iC-i,m)- 

: — 1 : — m 5 - — 1 - — m ' 

If, for example, we write R* m in the form 

T L (Xi, . . .,*„)(=: < Tfl^+i, . . . ,X m )(=: max{^/< J^ +1 ), 

then Tr involves more than one argument and Theorem 1 cannot be applied. 
However, our numerical studies show no evidence against (27). 

4.3. An example. We illustrate the proposed methods using a 2 4 facto- 
rial experiment from [10], pages 246-254, which investigates how tempera- 
ture, pressure, concentration of formaldehyde and stirring rate influence the 
filtration rate of a chemical product. The results are presented in Table 1. 
Column 2 of Table 1 gives the eight effect estimates with largest absolute val- 
ues and Column 3 the corresponding squared statistics, while SV = Ya=i Xi 
equals 15.11 for the seven effect estimates with smallest absolute values. 
Test statistics W u , m and W m -x, m are presented in the next two columns for 
v = 7. The SUF procedure identifies four largest active effects, irrespective 
of the two ways of choosing thresholds (d* m or d\, m ), while the SUS proce- 
dure identifies five largest active effects, also irrespective of the two thresh- 
old selections. For the sake of comparison, a step-down procedure from Voss 
and Wang [14], which uses test statistics Tsn,m = X m / min{0.92SV, 0.235n}, 
identifies three largest active effects. Finally, a MATLAB program for the 
evaluation of cutoff points is available from the authors. 
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Table 1 

The Montgomery [10] data, the step-up tests with fixed scaling (SUF) and the step-up 
tests with sequential scaling (SUS), and the related cutoff points 





Effect 




Test statistics 


The cutoff point 


3 at level a 


= 0.05 


m 


estimate 




Wu,rn 


Wm — l,m 




J* 




J* 

m — l,m. 


8 


-2.625 


6.89 


3.2 


3.2 


14.9 


14.9 


14.9 


14.9 


9 


3.125 


9.77 


4.5 


3.6 


26.5 


28.0 


16.4 


16.7 


10 


4.125 


17.02 


7.9 


4.8 


38.4 


42.0 


16.0 


16.3 


11 


9.875 


97.52 


45.2 


20.0 


52.2 


58.5 


15.5 


15.7 


12 


14.625 


213.89 


99.1 


16.1 


67.7 


77.5 


15.1 


15.2 


13 


16.625 


276.39 


128.0 


9.2 


85.0 


99.1 


14.6 


14.8 


11 


-18.125 


328.52 


152.2 


6.7 


104.5 


124.1 


14.3 


14.5 


15 


21.625 


467.64 


216.7 


(i.8 


126.3 


123.4 


14.0 


13.9 



5. Simulation study. A limited simulation study was conducted to com- 
pare five testing procedures: step-up tests with sequential scaling (SUS and 
SUSI using cutoff points determined by (24) and (26), respectively), step-up 
tests with fixed scaling (SUF and SUFI using cutoff points determined by 
(19) and (26), respectively) and the Voss and Wang [14] step-down proce- 
dure (SD). The testing procedures were evaluated in terms of four measures: 
(1) the experimentwise error rate (EER), (2) the probability of correctly 
selecting the number of inactive effects (PCSN), (3) the probability of com- 
plete correct selection (PCCS) and (4) the expected fraction of active effects 
that are declared active (Power). The simulation was carried out for several 
choices of k. Since the results are similar, we only present the choice k = 15 
on six cases below, following Venter and Steel [11]: 

CI: 0eH OM , 0i 5 = s; C4: G H 0>8 , fa = ■ ■ ■ = 0i 5 = s; 

C2: (BGHo.u; /3i3 = 014 = As = *; C5: G iT 0) i 2 , 012+i = is, 1 < i < 3; 
C3: G F ,10, 011 = • • • = As = s; C6: G # ,lo, Pw+i =is,l<i<5, 

where s takes values from to 8 with a step of 0.02. Each independent 
sample consists of 15 observations, each from N(/3i, 1) for 1 < i < 15. 

Simulation results for PCCS are nearly the same as those for PCSN and 
hence are not reported. Figure 1 presents selected results for the other three 
evaluation measures for a = 0.05 and v = 7, although findings are similar 
for other choices of a and v. Each point was determined based on 100,000 
simulations. In summary, all procedures control the EER. Second, there 
is a very small difference between the two ways of choosing cutoff points, 
especially between SUS and SUSI. Third, in CI, C2 and C5, the SUS is 
clearly the best. In C4 and C6, there is a small difference between SUF and 
SUS. In C3, the SUF performs better at small s, but the SUS is better at 
large s. The SD seems to be the worst in most selected cases. 
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C1 



C3 



C5 




o L ' ' ' J 

2 4 6 8 



C2 




2 4 6 



0-04 
0.02 




1 -^^^^ 1 

2 4 6 8 



C3 




2 4 6 



C3 




2 4 6 8 



C4 




2 4 6 8 




2 4 6 




2 4 6 



C5 




2 4 6 8 



C6 




2 4 6 



Fig. 1. Selected simulation results for five test procedures (SUSI, SUS, SUFI, SUF and 
SD) using a — 0.05, v — 7, three evaluation measures (EER, PCSN, Power) and six cases 
of parameter configuration (C1-C6) given in the text. 

6. Discussion. We search for active effects in orthogonal saturated de- 
signs by conducting simultaneous tests on a sequence of decreasing null 
hypotheses. A general class of level-a tests is provided for testing at least a 
certain number of active effects and the least favorable distribution is iden- 
tified to be the one at 3 . Two sets of simultaneous tests are derived with 

'- — m 

increasing rejection regions and their experimentwise error rates are con- 
trolled at a in the strong sense. Between these two sets of tests, the step-up 
tests with sequential scaling are recommended because our simulation study 
indicates that {R m -i m }m=v+i nas greater power in most cases. Since the 
maximal type I errors for {R* m } m=u+ i and {R^-i m } m=u+ i at m = u + 1 
and k are equal to a, simply enlarging the rejection regions cannot yield 
valid level-a tests. 

We can show that Lemma 1 is also true if U ~ i*i, n , an F-distribution with 
1 and n degrees of freedom, and V ~ -Fi >n (A), a noncentral F-distribution 
with 1 and n degrees of freedom and noncentrality parameter A. Conse- 
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quently, Theorem 1 remains true if we let X\, . . . be the order statistics 
of independent random variables with noncentral F distributions Fi jn (Xi), 
1 < i < k. This implies that our step- up simultaneous tests also work for 
squares of independent t-statistics. 
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